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Abstract: Learning Analytics is an emerging research field and design discipline that occupies the 
"middle space" between the learning sciences/educational research and the use of 
computational techniques to capture and analyze data (Suthers & Verbert, 2013). We propose 
that the literature examining the triadic relationships between epistemology (the nature of 
knowledge), pedagogy (the nature of learning and teaching), and assessment provide critical 
considerations for bounding this middle space. We provide examples to illustrate the ways in 
which the understandings of particular analytics are informed by this triad. As a detailed worked 
example of how one might design analytics to scaffold a specific form of higher order learning, 
we focus on the construct of epistemic beliefs: beliefs about the nature of knowledge. We argue 
that analytics grounded in a pragmatic, socio-cultural perspective are well placed to explore this 
construct using discourse-centric technologies. The examples provided throughout this paper, 
through emphasizing the consideration of intentional design issues in the middle space, 
underscore the "interpretative flexibility" (Hamilton & Feenberg, 2005) of new technologies, 
including analytics. 

Keywords: Learning analytics, epistemology, pedagogy, educational assessment, discourse 
analytics, social learning analytics 

1. INTRODUCTION 

Assessment is one area where notions of truth, accuracy and fairness have a very practical 
purchase in everyday life. (Williams, 1998, p. 221) 

Assessment sits at the heart of learning, but is hugely controversial. This is directly relevant to learning 
analytics, because — we argue — learning analytics implicitly or explicitly promote particular 
assessment regimes. Presently, many education systems are predicated on assessment regimes seeking 
to accredit knowledge and skills gained by students through formal assessments — often exam based. 
Proponents of such exams suggest they are the fairest way to assess competence and learning under 
controlled, reliable conditions. Assessment, pedagogy and curriculum are fundamentally related (Harlen, 
2007), but many regimes of what has come to be termed "high stakes" testing are criticized. For 
example, standardized assessments, including the Programme for International Student Assessment 
(PISA), American Standardized Assessment Tests (SATs), and English National Curriculum Assessments 
(Sats) face myriad problems. Not least among these is that the exams are criticized comprehensively 
(e.g., Davis, 1999; Gardner, 2011; Hopmann, Brinek, & Retzl, 2007) on two key validity criteria: for failing 
to represent adequately the types of problems people are likely to face in their everyday lives (external 
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validity), and that they fail to represent an adequate conceptualization of what it means to know — of 
what knowledge is (internal validity). The latter claim is that while assessments clearly measure 
something, a good grade does not necessarily reflect mastery (Davis, 1999). These fundamental issues 
are highlighted in a significant body of research (e.g., Davis, 1999; Gardner, 2011; Hopmann et al., 2007), 
and one of the objectives in writing this paper is to clarify the implications of these issues for the 
Learning Analytics community. 

In this paper, Section 2 considers the relationship between assessment systems and the sorts of 
epistemic challenges students might encounter. Section 3 introduces the concept of epistemic beliefs, 
and Section 4 goes on to discuss the relationships between learning analytics, and the triad of 
epistemology, pedagogy, and assessment. Section 4.2.1 then introduces pragmatic, socio-cultural 
approaches to learning analytics, which we suggest are well placed to probe or assess facets of learning 
that other learning analytics may not adequately address. To exemplify this argument, we draw a 
parallel between the psychometric measurement of epistemic beliefs and high stakes testing regimes 
(Section 5). The parallel is expanded in Section 6, which offers a detailed example of how our own 
orientation might make design decisions in the Learning Analytics "middle space." We conclude by 
reflecting on some broader ways in which these considerations might play out in various analytic 
approaches, maintaining the significance of the triad throughout. 

2. WHY WORRY ABOUT EPISTEMOLOGY? 

A primary concern of this paper is the relationship between epistemology, pedagogy, and assessment. 
Epistemology is the philosophical study of what knowledge is, and what it means for someone to 
"know" something. Central to the field of epistemology are questions regarding the nature of truth, the 
nature of justification, and types of knowledge, e.g., knowing how (skills) or knowing that (facts). 
Whatever "knowledge" is, "it is uncontroversial, pre-philosophically, that education aims at the 
imparting of knowledge: students are educated in part so that they may come to know things" (Siegel, 
1998, p. 20). Thus, pedagogy may in part be seen as the study of how to impart this knowledge to 
students — the science and development of approaches to teaching and learning for knowledge. 
However, epistemology's relationship to the more familiar concepts of pedagogy and assessment is a 
topic of educational debate (Davis, 1999; Dede, 2008; Kelly, Luke, & Green, 2008; Williams, 1998), and 
we will consider this in relation to learning analytics throughout this paper. 

Harlen (2007) depicted a triadic relationship between pedagogy, assessment, and practice. Influenced 
by this, and Katz's (2000) description of "competency, epistemology and pedagogy: curriculum's holy 
trinity" we depict the triad as in Figure l! 

In this picture, epistemology could be seen as driving assessments aimed at uncovering student 
knowledge, and driving pedagogy to build high quality knowledge to that end. In this view, assessment is 
targeted at the learning of high-level knowledge — it is assessment for learning. However, these 
relationships are not fixed; neither pedagogies nor epistemologies necessarily entail the other (Davis & 
Williams, 2002) (although they may implicate). Furthermore, as we will explore in this paper, 


1 We could also introduce the notion of "folk psychology" as a mediating factor between a teacher's views on 
knowledge and pedagogy; for example, if we hold that some (particular) children will never learn x, we are 
unlikely to attempt to teach it (a pedagogical "move") regardless of our epistemological stance regarding the 
nature of "x" (Olson & Bruner, 1996). However, in that paper, Olson and Bruner implicate epistemology in a 
number of their points regarding "folk pedagogy." 
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assessment is always concerned with devising proxies for "knowledge," around which there are 
philosophical (epistemological) and methodological issues. Some epistemological stances hold that it is 
not possible to "map" the knowledge that students hold onto their responses in assessments in reliable 
and valid ways. This issue is further confounded by the methodological limitations of all assessment 
methods, and by extension learning analytics. The adequacy of analytic techniques to illuminate learning 
should be open to enquiry. 



Figure 1. The Epistemology-Assessment-Pedagogy triad 

The situation is, therefore, a complex one — which facet of the triad has primacy over the others is not 
clear in either theory or practice, and may be dynamic according to need and circumstance. However, 
relationships between the three can certainly be identified, and throughout this paper, we draw out 
some of these with respect to learning analytics — which may be conceptualized as a component of 
assessment. Furthermore, we suggest that, although the relationship may not be a necessary one, 
assessment regimes do implicate particular epistemological stances. Our position is that attention to this 
middle space — bounded by epistemology, assessment, and pedagogy — provides key considerations 
for the design of learning analytics. 

Consider the following example from Denmark to illustrate the argument that, implicitly or explicitly, 
epistemological assumptions fundamentally shape pedagogy and assessment, and hence, the kinds of 
learning analytics that one deploys to achieve those ends. In Denmark, a pilot project was conducted 
permitting the use of the Internet (but not communication sites) to support students in five school- 
leaver subject exams. 2 This made it possible to set questions requiring the use of multimedia and 
individual Internet searches where students may be given unfamiliar resources and permitted to source 
information for themselves from the Internet. For example, a student might be asked to write about a 
poet whom they have not studied (preventing rote learning), having been provided with a poem, plus 
another by a contemporary, a short biography, and perhaps an image from the time. Thus, while Danish 
students are expected to evidence "knowledge-that" — knowledge of facts — they must also exhibit a 
higher level of "knowing-how," for example around information processing, synthesis, and 
metacognitive abilities — which remain unassessed in countries restricting access to external resources 
that might enhance the student's capability. While this, of course, remains a controlled assessment 
context, the example illustrates how even within a system reliant on exams, those exams might be 
conducted on a rather different epistemological grounding. Assessment regimes such as the Danish 
example may be taken to reflect a holistic epistemology in which how one comes to know is as 
important as what one comes to know, and in which it makes little sense to pick out individual tokens of 
knowledge in decontextualized ways (Davis & Williams, 2002; Davis, 1998, 2005; Katz, 2000). 


2 Steen Lassen (a Danish Education Minister) on the piloting of Internet access in exams: 
http://vimeo.com/8889340 subsequently adopted by some Danish universities (Cunnane, 2011). 
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We can contrast such assessments with high stakes testing regimes whose construct validity and 
external validity have been questioned. For instance, Davis (1999; 2006) argues that such instruments 
neither assess those facets of learning they set out to test, nor those facets of learning that would likely 
be utilized in the everyday deployment of knowledge in any particular domain. Davis has argued that 
high stakes testing is inadequate for understanding learning, in so far as its construal of that learning is 
necessarily restricted by a desire for highly reliable metrics of success. As such, it must exclude the 
nuanced understanding of student meaning-making, the social context in which learning occurs, and 
how knowledge is constituted and enacted. He argues that this, as opposed to acquisition, is the 
appropriate way to talk about knowledge. Davis draws on notions of situated cognition (Salomon, 1996) 
and socio-cultural approaches (Saljo, 1999) — particularly Saljo's "Literacy, Digital Literacy and Epistemic 
Practices: The Co-Evolution of Hybrid Minds and External Memory Systems" (Saljo, 2012). Saljo 
highlights that: 

From the learning and literacy points of view, such tools [memory aides and knowledge 
management systems of various sorts] imply that users' knowledge and skills, as it were, are 
parasitic on the collective insights that have emerged over a long time and which have been 
entered into the instrument in a crystallized form: algorithms, grammatical rules and 
concepts, etc. The user will manipulate the artificial memory system in a number of ways in 
order to see what comes out of the processing that goes on in the machine. (Saljo, 2012, p. 

14) 

However, 

Engaging with external memory systems thus requires familiarity with a varied set of 
epistemic practices that range from deciphering letters on a page through familiarity with 
meaning-making in relation to discourses and genres of texts and other media, to meta¬ 
knowledge about how such resources may be used. (Saljo, 2012, p. 12) 

Saljo makes an epistemological claim, specifically, a socio-cultural, pragmatist claim: that there are 
important literacies and practices to be mastered in learning; that those should themselves be objects of 
assessment; and language and discourse are critical filters on our grasp of the world. Such an 
epistemology has implications for how we teach, what we assess, and which analytics techniques might 
be deployed. "Success" can no longer be defined as a matter of regurgitating, unaided, the correct 
information in a two-hour exam. Such an epistemology also, we argue, offers a perspective on why even 
technologically advanced societies such as Denmark that assess knowledge in less abstracted, socially 
embedded ways, information seeking and processing via the Internet and search engines remains a 
significant challenge for students (UAGU, 2010, p. 15) Although the Internet provides wider access to 
information, information is not knowledge. Student engagement with information should consider both 
the kinds of knowledge that we might call transferable competencies or skills — including those higher 
order skills often known as metacognitive abilities — and more propositional or fact-based knowledge. 
In this context, we might consider information management and seeking not only as a means to an ends, 
but as a way to encourage interaction with a complex network of information. As argued by Tsai, as not 
only: 


...a cognitive tool or a metacognitive tool; rather, it can be perceived and used as an 
epistemological tool. When the Internet is used as an epistemological tool for instruction, 
learners are encouraged to evaluate the merits of information and knowledge acquired 
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from Internet-based environments, and to explore the nature of learning and knowledge 
construction. (Tsai, 2004, p. 525) 


In this conception, learners are encouraged to think about the context, reliability, validity, certainty, and 
connectedness of knowledge. 

To summarize, this section has argued that a consideration of epistemology is important to learning 
analytics in two related senses: 


• The ways that we assess, the sorts of tasks we set, and the kinds of learning we believe to take 
place (and aim for) are bound up in our notions of epistemology. Learning analytics are not 
objective or neutral: data does not "speak for itself" but has been designed by a team who, 
implicitly or explicitly, perpetuate the pedagogical and epistemological assumptions that come 
with any assessment instrument. 

• The Danish example shows concretely how epistemology relates to assessment regimes. When 
knowledge is seen as something that can only be evidenced in contextualized activity, and when 
it is embedded in one's physical and digital environment, the role of the Internet is redefined as 
a metacognitive tool that cannot be excluded in assessment. 

These epistemological considerations foreground the quality of a student's enquiry processes as 
important, not just whether they get the right answer. An aspect of that enquiry process is epistemic 
beliefs — and it is to these that we will turn shortly. 

Before we do so, we should note the importance of the whole triad in bounding the middle space of 
learning analytics, as defined by the programme chairs of the 2013 learning analytics conference: 

In summary, although individual research efforts may differ in their emphasis, we believe 
that all research in Learning Analytics should address the "middle space" by including both 
learning and analytic concerns and addressing the match between technique and 
application. Advances in learning theory and practice are welcome, provided that they are 
accompanied with an evaluation of how existing or new analytic technologies support such 
advances. Advances in analytic technologies and methods are welcome, provided that they 
are accompanied with an evaluation of how understanding of learning and educational 
practices may be advanced by such methods. (Suthers & Verbert, 2013, p. 2) 

As noted earlier, it is our claim that epistemology is fundamental to the understanding of assessment, 
and that learning analytics fits in this scope. Furthermore, there is a bi-directional influence. Thus, 
design choices around assessments, often practical ones, may push towards particular epistemological 
stances, for example in the case of high stakes testing. In the other direction, sometimes explicitly 
epistemological stances, as in the Danish example, lead to particular forms of assessment being 
privileged. The role of learning here is also important; we have noted that the focus of pedagogy can be 
taken to be the bringing about of knowledge in students, as conceptualized by epistemology. However, 
practical considerations are again in play here, including the particular sorts of information privileged. 
Particularly pertinent is the role that we intend learning analytics to play in assessment: learning 
analytics that provides summative feedback is likely to relate to different types of learning, analytics, 
and outcome than learning analytics based on the provision of formative feedback. 
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The "interpretative flexibility" of new technologies, including analytics, is high: when we consider 
appropriation of technology within particular social settings, we should be mindful of not falling into the 
trap of technological determinism (Hamilton & Feenberg, 2005). As Crook and Lewthwaite (2010) note, 
our expectations for technology for transformative change should be mitigated by an understanding of 
those technologies in wider educational systems. Moreover, we should understand that technology's 
influence comes about through pedagogic change, not out of technology's direct effects (Crook & 
Lewthwaite, 2010). Indeed, the epistemological stance taken in this paper is one well aligned with 
approaches that situate technology in its wider application, in non-deterministic ways (Oliver, 2011). 

As noted above, the adequacy of analytic techniques to meet learning should be open to enquiry. This 
involves understanding the design trade-offs one is negotiating (intentionally or unintentionally) in 
developing and deploying technological solutions. This paper argues that this analysis — this bounding 
— fundamentally involves understanding epistemology, pedagogy, and assessment and their 
interrelationships in the context of learning analytics. In Section 6 we return to these considerations in 
the context of the example introduced below — that of epistemic cognition. 

3. EPISTEMIC BELIEFS 

One facet of students' dynamic interaction with the world of information relates to how they 
conceptualize the information needed to answer a question — their epistemic beliefs regarding the 
nature of the question and how it may be answered. The sorts of assessment, and pedagogy, to which 
students are exposed will shape the types of epistemic challenge they encounter in their education — 
systems with a focus on "right answerism" and limited access to external epistemic resources offer 
fewer opportunities for challenging knowledge claims (Davis, 1999; Katz, 2000). This paper thus 
differentiates two related concepts: 

1. Epistemology, which we introduced above, relates to the philosophical analysis and 
conceptualization of curriculum content and assessment for knowledge 

2. Epistemic Beliefs, which we now introduce, relates to the intrapersonal, psychological 
conceptualizations that individuals hold regarding knowledge 

Indeed, a key component of formative Assessment for Learning (AfL) may be the disambiguation of the 
epistemic requirements of questions — in terms of understanding the question, its context, and the 
knowledge required to answer the question (Black & Wiliam, 2009). That is, AfL may play a crucial role in 
guiding a student's epistemic beliefs on the subject of the epistemological assumptions built into 
assessment systems themselves. 

Table 1 indicates four dimensions of epistemic beliefs for which there is general agreement across the 
various models of belief. 3 These dimensions are useful to consider in relation to student understanding 
of knowledge domains. For example, in the context of search engine tasks, "epistemological beliefs are a 
lens for a learner's views on what is to be learnt" (Bromme, Pieschl, & Stahl, 2009, p. 8). In such tasks, 
student search activity may be analyzed using the dimensions in Table 1 (e.g., Mason, Ariasi, & Boldrin, 
2011), providing a lens onto students' understanding of their own learning, their task demands, and how 
they meet those demands. 


3 See, for example, Schraw (2013) for a review of the multiple theoretical frameworks. 
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Epistemic beliefs are thus one example of the type of construct that learning analytics may probe; 
however, they are also a particularly good example given their relationship to our everyday dealings 
with the world of information, and their relationship to pedagogy, assessment, and classroom practices 
(Hofer, 2001). Section 5 will discuss epistemic beliefs in relation to their measurement, but we shall first 
introduce some established approaches to pedagogy. 


Table 1. Dimensions of epistemic belief (adapted from Mason, Boldrin, & Ariasi, 2009, p. 69) 


Dimension 

Description 

Certainty of knowledge 

The degree to which knowledge is conceived as stable or changing, 
ranging from absolute to tentative and evolving knowledge 

Simplicity of knowledge 

The degree to which knowledge is conceived as compartmentalized or 
interrelated, ranging from knowledge as made up of discrete and 
simple facts to knowledge as complex and comprising interrelated 
concepts 

Source of knowledge 

The relationship between knower and known, ranging from the belief 
that knowledge resides outside the self and is transmitted, to the 
belief that it is constructed by the self 

Justification for knowing 

What makes a sufficient knowledge claim, ranging from the belief in 
observation or authority as sources, to the belief in the use of rules of 
inquiry and evaluation of expertise 


4. OUR LEARNING ANALYTICS ARE OUR PEDAGOGY 

Buckingham Shum (2012) used the shorthand "our learning analytics are our pedagogy" to draw 
attention to the arguments set out in more detail above: that the types of analytic we chose to deploy, 
and the ways in which we deploy them implicate particular approaches to learning and assessment. This 
is particularly important given that any use of analytics will be in the context of a wider educational 
ecosystem, as Crook and Lewthwaite (2010) noted. It is this relationship between the types of analytic 
we deploy and our pedagogies that we now consider. 

4.1 Pedagogy and Analytics 

The relationship between learning analytics and pedagogy is important because they are both bound up 
in epistemology — what knowledge is. This section explicitly introduces the relationship between a 
number of established pedagogic approaches and learning analytics. These are not intended as 
comprehensive reviews, but rather as brief overviews of how the relationship between pedagogy and 
learning analytics might be conceptualized. The following section expands on some key ideas here, 
before moving on to explicate the core topic of this paper — a socio-cultural learning analytic — and 
one proposed instantiation of a learning analytic based on this approach. 

4.1.1 Transactional or instructionalist approach 

Transactional approaches hold that learning entails the transfer of knowledge from the knower 
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(teacher) to the learner (student). They are characterized by a perspective on assessment in which 
success is "out there," assessable in the degree of correspondence between the claims that learners 
make, and the facts that they have been taught. Clearly there is a role for mastering facts in many 
curricula, and the technological ease with which these may be automatically assessed underlies their 
current dominance of learning analytics and automated assessment (formative and summative). 

Analytics Implications: Learning analytics based on transactional approaches will tend to focus on simple 
metrics such as test scores, not requiring deeper analysis of more complex artefacts, or the processes by 
which they were derived. 

4.1.2 Constructivist approach 

Constructivist models focus on those forms of learning that occur in the learner's guided exploration of 
and experimentation with the world, typically in classrooms or online environments. Constructivist 
models are likely to measure success as quality of construction, with learners experimenting with their 
environment, and being capable of using tools that are appropriate for their given age. 

Analytics Implications: Learning analytics with a constructivist focus will focus on progress, particularly 
through tracking and judging the modifications made to a set of materials, resources, or tools selected 
and arranged by the educator. An example of analytics in this tradition would be tracking the evolution 
of digital artefacts within the Scratch visual programming environment and community (Maloney, 
Resnick, Rusk, Silverman, & Eastmond, 2010). 

4.1.3 Subjectivist or affect-based approach 

Subjectivist perspectives can be characterized as de-emphasizing learning qua academia, with more 
attention to personal affect. While individual affect is a concern for educationalists, it is rarely if ever the 
overarching concern in the consideration of learning. One context in which affect is important is learning 
in complex socio-technical challenges: while there are certainly better and worse answers, there is too 
much information and no known best solution. Information seeking in such contexts can draw on 
subjectivist approaches that measure whether the user is "satisfied" with the information they have 
found. Another context in which self-reporting is an important proxy for learning is research into 
dispositions (Deakin Crick, Broadfoot, & Claxton, 2004) and "mindsets" (Dweck, 2006) — learner 
willingness to engage with opportunities that will challenge them, or stretch other transferable 
competencies such as questioning or collaborating. 

Analytics Implications: In tandem with other approaches, learning analytics based on subjectivist 
approaches are likely to provide motivation assessments for understanding why someone is (or is not) 
engaging in particular actions. Such analytics may focus on self-reporting through survey tools 
(Buckingham Shum & Deakin Crick, 2012) or affect-based semantic mark-up such as blog tagging 
(Ferguson, Buckingham Shum & Deakin Crick, 2011), alongside automated approaches such as textual 
sentiment analysis. 

4.1.4 Apprenticeship approach 

Apprenticeship approaches are sometimes used in learning analytics with an interest in whether the 
learner has become part of a community of practice or enquiry. In this view, success is about "being part 
of" a given group; it is bound up in notions of communities of practice — that "to know x" is to act 
towards x in some way defined by (or reflected in) the behaviours of some community or other. 

Analytics Implications: Analytics based on apprenticeship approaches are likely to focus on classifying 
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expert and novice users, and the shift from novice to expert. Such analysis may explore behavioural 
markers that mirror those made by "experts," but may not explore the reasons or meanings implicated 
in such moves. Epistemic Network Analysis of user data from gaming environments is designed to 
quantify the degree to which learners demonstrate behaviours valued in a professional community 
(Shaffer et al., 2009). The creation of social capital might be considered another proxy for community 
membership, overlapping with the next category. 


4.1.5 Connectivist approach 

Connectivism claims to highlight a perspective on epistemology that translates into a learning analytics 
framework. Within this view, learning is about understanding how to connect ideas appropriately, and 
where to find such information. The suggestion is that in the case of the connectivist knower "the act of 
knowing is offloaded onto the network itself" (Siemens, 2006, p. 33). Within this perspective then, 
success is about building connections between ideas. 

Analytics Implications: Connectivist approaches use network analysis to explore the "connectedness" of 
a learner's knowledge — in terms of both concepts, and social connections. Analytics would look at how 
networks' size, quality, and changes over time can serve as proxies for effective learning (Dawson, 2010; 
Haythornthwaite & de Laat, 2010). 


4.1.6 Pragmatic, socio-cultural approach 

Pragmatic approaches (building on, for example, Dewey, 1998) hold that learning occurs in the 
development and negotiation of a mutually shared perspective between learners. Pragmatists suggest 
that, as human knowers, our conception of some given thing is bound up in our understanding of its 
practical application — and that is all. When we attempt to understand truth beyond such a 
conceptualization of practical activity, we are likely to fail. Thus, success is in use — the measure of 
success is how useful the information is for the purposes it is employed; it is socio-culturally embedded 
and mediated, and may be in flux as activities are defined and redefined. 

Analytics Implications: Pragmatic approaches have traditionally focused less on assessing the products 
of learning (except where they are being used for something), and more on the process. Analytics tools 
in socio-cultural approaches encourage learners to reflect on their own activity, in an attempt to 
understand how they can develop their skills in information processing, in their own particular contexts. 
Analytics within this approach might attend particularly to quality of discourse for learning, for creating 
a mutuality of perspectives (Edwards & Mercer, 1987) including in collaborative information seeking 
tasks (Foster, 2009; Hertzum, 2008; Lazonder, 2005). Our previous work is in this tradition, drawing on 
socio-cultural discourse analysis (Mercer & Littleton, 2007), and emerging conceptions of the pragmatic 
web (Buckingham Shum, 2006). This research foregrounds how students interact with information, 
make sense of it in their context and co-construct meaning in shared contexts. These are on-going 
processes that highlight the question of how learning analytics fits into the context of AfL and pedagogy. 

Having summarized the sorts of relationships we might see between pedagogical approaches and 
learning analytics, let us turn to epistemology. 

4.2 Epistemology and Analytics 

The stance we take with regard to the relationship between epistemology, assessment, and learning 
analytics relates to the issue of whether we place analytics in the role of diagnosis or a kind of 
biofeedback. Is learning analytics (and assessment) serving as the end-point of, or a component of 
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pedagogy? As a diagnostic, we seek to accredit learning through defining behavioural proxies taken as 
evidence of knowledge and competencies. As biofeedback, learning analytics is used to support learners 
in their own self-regulated learning activities, giving them feedback on changes they make and their 
impact on learning outcomes, but without, generally, making strong evaluative judgments regarding 
such changes. The former is thus more closely aligned with assessment of learning — often instantiated 
in high stakes summative assessment — while the latter is closer to assessment for learning — in which 
assessment is a continuous process through which formative feedback may be given to further develop 
student learning (see, for example, Black & Wiliam, 2001; Gardner, 2011). If evidencing process-centric 
competencies is defined as part of the summative assessment criteria, then the two categories 
converge. For example, the process competencies of evidencing sound argumentation in discourse, 
higher resilience when stretched with new challenges, or the creation of social capital within a 
community of practice, might conceivably be assessed summatively through analytics. 

The relationships highlighted in sections 4.1.1 to 4.1.6 serve as general pointers to the sorts of 
relationships we might see between pedagogy and learning analytics. There we also highlight views on 
learning, alongside notions of how success may be defined within these approaches; that is, when these 
systems might accredit knowledge to the student. Fundamentally, accreditation implicates 
epistemological stances regarding when knowledge may be claimed (or not). The preceding analysis 
suggests roles for learning analytics in accrediting mastery in three senses: 

1. Mastering curriculum content: This is the dominant focus of analytics approaches at present, 
seeking behavioural markers using e-assessment technologies of varying sophistication, in order to 
generate summaries at varying granularities, for both individuals and cohorts. (See transactional and 
some constructivist approaches.) 

2. Evidencing membership and processes: This approach to learning analytics looks for behavioural 
proxies that indicate a student is part of a particular subgroup; positive feedback is given towards 
moving students into "successful" subgroups, but little attention is paid to the qualities of those 
groups except instrumentally. (See affect-based, apprenticeship, and possibly connectivist 
approaches.) 

3. Success is use: This approach looks for students developing personal and collective representations 
of curriculum content and engagement in sensemaking about not only this material, but also their 
own analytics. One characterization of this family of approaches has been as Social Learning 
Analytics (Buckingham Shum & Ferguson, 2012). (See connectivist and pragmatist approaches.) 

These three broad conceptualizations of learning analytics relate to the issue of whether or not we are 
deemed to consume, discover, or create knowledge — is it "out there" for us to take, do we need to 
investigate to find it, or is it emergent from the contexts in which that knowledge is applied and reified 
in activity? This is not a claim about learning or pedagogy, but a related claim about the status of 
knowledge and its assessment. We discuss this further in section 6.4 with reference to a particular 
example. 

4.2.1 Pragmatism and socio-cultural definitions of "context" 

The nuance of claims surrounding epistemology and assessment is important. In the introduction we 
referred to research arguing that conventional exams are designed to maximize the reliability of results, 
at the cost of straitjacketing what can be defined as learning (poor internal or construct validity) and 
thus what constitutes evidence of learning (poor external validity). Moreover, if we are to argue that 
individual tokens of knowledge cannot be identified (and "owned"), then we should accept that "the 
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content of a specific item of knowledge depends in part on how it is related to other knowledge" (Davis, 
2006). Thus, socio-cultural setting, interaction, and the purposes for which any artefact or knowledge — 
in the broadest sense — is being used are all of fundamental importance in understanding how people 
make meaning and learn. Contextual sensitivity is thus a key facet of pragmatist approaches. 

Pragmatic approaches, broadly, are likely to focus on the dynamic nature of information needs, and the 
discourse and other artefacts that mediate our relationship with information in the world. It is not a 
postmodern approach, in the sense that postmodern approaches take either a relativist approach (there 
is no fixed truth) or a normative one (the dominant theme is correct at that time) to knowledge, but 
rather one that focuses on use, and meaning, over accreditation of facts to things in the world. 

Given the salience of context in this approach, it deserves further explication. As with learning analytics 
generally, context may be taken as very mechanistic; for example, the claim that a person in 
place/course/role/ability band "x" should see resource "y," or other approaches that would include 
time, topic, or social-group resource discovery. No doubt, some of these features will prove useful, and 
indeed the use of semantic web technology in social learning analytics (Ferguson & Buckingham Shum, 
2012) may be particularly interesting. However, in addition to computational, context is often defined in 
terms of temporal, linguistic, aptitude, and geo-spatial metadata. In discourse, however, we draw 
attention to the following: 

1. We emphasize the discourse in which, and through which, context is constituted (Edwards & Furlong, 
1978; Potter & Edwards, 2003). That is, we take the discourse to have a multifaceted role in 
constituting, and helping learners make sense of, the context. 

2. Discourse is fundamentally associated with the sense-making that occurs in respect of any particular 
task being undertaken; the use being targeted is fundamental to context. Stark examples highlight 
this importance; for example, when we ask students to critique versus summarize a paper, we 
expect rather different outcomes. Assessment regimes that make this explicit may facilitate the 
design of analytics that capture the context of "doing x for purpose y." 

3. Assessment regimes, and the broad range of tools, technological and otherwise used by learners, 
also act as mediating artefacts impacting how people perceive their task and its solution, which all 
shape the context of use. 

A conception of learning analytics that recognizes the importance of context is central to epistemic 
beliefs: 

A sophisticated epistemology entails context-sensitive judgements. Thus, they point out 
that it is not very sophisticated to view the idea that the earth is round rather than flat as 
"tentative" whereas theories of dinosaur extinction do require a more tentative stance. 
(Barzilai & Zohar, 2012, p. 42) 

Similarly, building spurious connections between ideas in an attempt to evidence a "complex" view of 
knowledge is less sophisticated than demonstrating the need for moderation, and so on. Context is thus 
key to understanding epistemic beliefs, the analysis of which seems highly suited to the biofeedback 
approach to formative assessment analytics, introduced earlier. 

The next section further expands this claim in the context of psychological assessment of epistemic 
beliefs, firstly in "mainstream" psychological approaches, and then that of the discursive approach, 
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which similarly holds context and discourse to be fundamental to understanding thinking. Section 6 then 
returns to learning analytics, drawing out the relationship between analytics, and the measurement of 
epistemic beliefs in our illustrative example for socio-cultural, pragmatic analytics. 


5. MEASURING EPISTEMIC BELIEFS 


The complexity of epistemic cognition suggests a particular perspective on how we are to understand 
these beliefs. No approach "mirrors" reality with a true, immutable, incontrovertible perspective on a 
learner's epistemic cognition. This concern is a dual one. Firstly, it is a methodological concern regarding 
our access to the world, our ability to "get at" what is out there. Secondly, it is a conceptual and 
psychological concern, regarding the nature of epistemic cognition and whether it itself is stable — 
developmental^, and across domains — or shaped in some way by resources or beliefs. These two 
concerns are reflected in the epistemic beliefs literature. Firstly, cognitive developmental models (King 
& Kitchener, 2004; Kuhn & Weinstock, 2002) suggest that individuals progress through a sequence of 
increasingly sophisticated epistemic beliefs, while multidimensional perspectives (Flofer, 2001; 
Schommer, 1990) suggest that epistemic beliefs can be separated into dimensions, within which levels 
of sophistication can be identified (Greene, Muis, & Pieschl, 2010, p. 248). Flowever, both of these 
assume a fixed, uni-directional developmental trajectory, where beliefs are seen as global across (and 
within) domains. The resources view, in contrast, emphasizes the interaction of believer, with resources, 
highlighting that at various points in any task a cognizer may invoke differing resources (Flammer & Elby, 
2003). 

Secondly, methodologically the developmental models have tended towards interviews and laboratory 
tasks, while multidimensional models have emphasized paper and pencil self-report measures 
(DeBacker, Crowson, Beesley, Thoma, & Flestevold, 2008). Both of these approaches reflect the fixed 
perspective on beliefs from which theory they stem. Importantly, although three major survey 
instruments have been developed and deployed, — including in search engine tasks (Lin & Tsai, 2008; 
Schommer, 1990) — they are heavily criticized for their psychometric properties (DeBacker et al., 2008). 
Furthermore, while some studies have used interviews (Barzilai & Zohar, 2012; Mason et al., 2009), 
think-aloud protocols (Barzilai & Zohar, 2012; Ferguson, Braten, & Strpmsp, 2012), or systematic 
observations (Scherr & Flammer, 2009) such methods may be limited in their insights, particularly where 
self-report data is to be used and interpreted by researchers. Importantly, they are also not appropriate 
for the study of online, collaborative, or geographically and temporally spread activities — in particular, 
online information seeking, or information processing more broadly. These approaches reflect the 
epistemology of current assessment regimes, as indicated in Section 2, and seem to implicate the view 
of "fixed" psychological constructs — whether intelligence or epistemic beliefs, as further discussed 
throughout Section 3. 

Building on sections 4.1.1 to 4.1.5, we can identify a number of analytic tools and their relationships to 
particular forms of data. Some forms of analytics rely on a belief that particular methods (self-report in 
particular) are: a) true reflections of reality, b) whole reflections of reality (i.e., they cover all the 
relevant ground), and c) probe "real" constructs. Flowever, while self-report measures may be useful 
particularly as discussion prompts with students, they are not necessarily the most useful approach for 
many purposes. In both assessment and psychological testing, they suffer from issues of validity (Section 
2). Thus, other learning analytics tools may prove more useful. 

That is not to write off such tools, which may be particularly useful in exploring population differences 
and providing basic metrics to distinguish between people (candidates for a job, for example). Rather, it 
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is to note that they do not align well with formative assessment (although some may be used for such 
purposes); they have a tendency to "teach to the test" — or in the case of psychometrics, to rely on 
tightly culturally bound constructs; they are often used to make claims about states of the world as 
"products" rather than an exploration of processes and wider contexts. 

In contrast, while those adopting a resources view of epistemic beliefs may also utilize such methods — 
in particular those involving think aloud and interview data — they also accord well with Osterholm's 
discursive stance, which suggests that we should not see beliefs and communication as "two separate 
'objects' that can affect each other, but as more integrated aspects of cognition and/or behaviour" 
(Osterholm, 2010, p. 242). The resources view describes "the activity, the discourse, as the site where 
epistemological beliefs come to existence, through explicit or implicit references to prior experiences 
(epistemological resources)" (Osterholm, 2009, p. 262). Osterholm's argument is that the resources 
perspective can be combined with Hammer and Elby's (2003) resources model. In this model, epistemic 
beliefs are not viewed as fixed or developing cognitive models ranging over one or more domains, but 
rather are seen as dependent upon the resources available to the cognizer at any time. This view of 
epistemic beliefs as "theory-in-action" — in which context, domain, culture, and task conditions interact 
— accords well with the idea that context is fundamental to understanding meaning. 

5.1 Trace Data for Epistemic Beliefs 

While Osterholm is primarily interested in spoken interactions, learning analytics may extend this 
interest into the exploration of users' interactions with artefacts. A tool for such analysis may come 
through the use of trace data, which is more or less implicitly created by the student. For example, 
Stadtler and Bromme (2007) analyzed the ways participants found, extracted, and moved information, 
which could be used to explore information about their beliefs (e.g., visiting few websites indicates trust 
in those sites visited; Greene et al., 2010). Importantly in this study, users were either given evaluation 
prompts regarding multiple documents in the medical domain or not; those who received such prompts 
subsequently recalled more facts and were better able to evaluate sources. If systems of prompts 
promote laziness, we should be concerned. Where they improve outcomes, however, analytics should 
explore the best ways to implement them effectively and sustainably to support high quality pedagogy 
and AfL. 

Furthermore, Greene et al. (2010) point out that many behaviours that would ordinarily be difficult to 
observe can be explicitly elicited in the context of Computer Based Learning Environments (CBLEs), for 
example: 

...participants who report belief in objective truth and omniscient authority may self- 
regulate quite differently than participants with a desire to evaluate multiple forms of 
justification. Likewise, participants who believe in the inherent subjectivity of all knowledge 
may, on average, select more representations than those who look for an objective truth. 
(Greene et al., 2010, p. 254) 

The claim is thus that epistemic beliefs will be brought to bear on knowledge tasks in ways that can be 
meaningfully captured, in particular using the digital trace data left as a "by-product" of engaging in 
tasks (such as issuing searches, browsing websites, or messaging peers), or through the intentional 
creation of digital artefacts whose evolution can be traced (e.g., documents, bookmark collections, 
concept maps). 
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Trace-based learning analytics provides a means to tackle the static, decontextualized view of epistemic 
beliefs instantiated by questionnaire methods, offering a more authentic perspective on epistemic 
action than experimental contexts. However, the limit on what can be inferred about the mind of a user 
solely from low-level system traces is a well known problem in human-computer interaction. While 
trace data is unobtrusive, it necessarily gives an incomplete picture. In particular, people may have 
reasons for some behaviours that cannot be probed using such data, ranging from epistemic (as 
discussed above, for example with regard to the "flat earth" issue), to practical (ICT failures), to 
pragmatic (the demands of the task place a short time restriction on the activity), and so on. In 
summary, even though analytics regarding epistemic beliefs may be — at best — a dirty lens onto those 
beliefs, when analytics are considered in action as a tool for sense-making, they may provide an 
insightful tool for learners to dissect their own metacognitive and self-regulatory behaviours, as well as 
for educators and researchers to study them. 

Trace data thus provides one means by which epistemic beliefs could be examined. However, trace 
could refer to many things, and as discussed in sections 4.1.1 to 4.1.5, the data collected may not 
represent an appropriate teaching epistemology, nor capture adequately student epistemologies (see 
section 2). The next section will discuss some learning analytics that may address this issue. An 
interesting notion then, is attempting to delve further into the sense-making significance behind 
particular semantic moves in a given environment. Thus, Greene et al. (2010) (see Section 5.1) described 
one method of trace analysis for epistemic beliefs built on information moves. Other examples of such 
trace capture could also be structured to gather student data in particular ways — some of which may 
be quite naturalistic (capturing search queries, or Facebook posts to explore "problems" encountered, 
or interactions made; De Liddo, Buckingham Shum, Quinto, Bachler, & Cannavacciuolo, 2011), and 
others of which might push students into information structuring activity in which they would not 
otherwise engage, such as argument mapping. 

However, in encouraging such structuring by learners, and claiming to capture information about what 
they are doing, some may argue that we are simply reifying the constructs we have set out to explore. 
That is, if we are interested in epistemic beliefs, and set up a system to push students to make epistemic 
beliefs explicit, it does not matter whether those students have underlying epistemic beliefs because the 
system forces them into making some (it makes them reify). While for psychologists who wish to 
uncover underlying beliefs using the kind of psychometrics described above this is problematic, we do 
not see this as a concern for our project, because in our discursive, socio-cultural, pragmatic approach 
the interest is in beliefs as "theory-in-action." In this view, the claim is not that the measurement of 
beliefs is not possible, but rather that when we take measurements, the discursive context is 
fundamental to the practices being observed, and the ways that the beliefs are instantiated in action. 
Thus, learning analytics provides a means to tackle the static, decontextualized view of epistemic beliefs 
instantiated by questionnaire methods, offering a more authentic perspective on epistemic action than 
experimental contexts. 

6. EXAMPLE: DISCOURSE-CENTRIC ANALYTICS FOR EPISTEMIC BELIEFS 

We have reviewed the ways in which the educational triad bounded by epistemology, assessment, and 
pedagogy defines critical aspects of the "middle space" between learning and analytics, which can lead 
to very different kinds of analytics. We now present a more developed example, to illustrate how the 
preceding discussion can translate into tangible design. 

Within a pragmatic socio-cultural perspective on the triad, use of knowledge, and an understanding of 
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its rich culturally embedded connectedness is fundamental. We have emphasized the role of context at 
various points throughout this piece, indicating in particular the need to understand discourse as 
created in, and a creator of, context. The perspective is thus fundamentally social in nature, taking 
knowledge to be discursive in nature, with learners benefiting in particular from timely formative 
feedback in which language is used to construct knowledge. 


Standardized assessments, just as psychometric tests, are good at population-level analytics. However, 
they are problematic at individual level analysis, at understanding change, and at understanding 
individual traits and capabilities. The perspective we have outlined in this paper suggests analytics may 
play an important role in assessment for learning, providing formative, individualized, feedback to 
learners and their teachers. 


In the following example, we exemplify such an approach; however, while we hope the example given is 
illustrative, it is not intended to be proscriptive nor indeed do we claim it is without faults; rather, we 
seek to illustrate how the stance we have taken here has been "brought out" in the analytics — how we 
have brought the learning and the analytics together, using the triad to bound that middle space. 

This example offers an illustration of how an analytic device might be used to offer formative feedback. 
In this case, our focus is not explicitly on the content level knowledge claims we can make regarding 
student learning, but rather on how students develop their understanding. The example offers a means 
to explore supporting student learning through formative feedback, and social learning of a discursive 
nature. Of course, further consideration should be given to the wider context of the activity and 
discourse we would wish to provoke. 

Our example comes from work at the Open University, based around the Cohere social annotation and 
knowledge mapping tool (Buckingham Shum, 2008) and subsequent work on its potential for socio¬ 
cultural discourse-centric learning analytics (De Liddo et al., 2011). Cohere is a web application for 
mapping ideas, concepts, and arguments that can be annotated directly onto source websites. Users 
enter ideas — nodes with meaningful classifications — and are then invited to "make the connection" 
with meaningfully labelled edges, to create a conceptual graph. Both ideas and connections may also be 
tagged, to add a further level of semantic data. Cohere is designed as a tool to enable users to build 
their own structures, but also to share these, and integrate the nodes and connections of other users, 
thus building up communities of enquiry around particular disciplinary topics. This deliberate design 
decision not to impose an ontology onto users makes it explicitly pragmatic, and discursive in nature 
(Buckingham Shum, 2006). In the context of learning sciences and CSCL research, Cohere is a Web 3.0 
(social and semantic web) era application exemplifying the knowledge-building paradigm that has 
inspired earlier reflective, knowledge structuring software tools (Novak, 1998; Scardamalia and Bereiter, 
1993; Suthers, Weiner, Connelly, & Paolucci, 1995). 

Thus, Cohere renders proxies of learner understanding in the form of a semantic network of textual 
nodes that may also link to multimedia resources. Three recent technical advances are also of note: a 
version called the Evidence Hub now feeds analytics back to end users as well as to system 
administrators, and enables the embedding of multimedia within nodes (De Liddo and Buckingham 
Shum, 2013), while the textual content of nodes can now be analyzed for significant patterns using a 
rhetorical parser (Simsek, Buckingham Shum, Sandor, De Liddo & Ferguson, 2013). Together, these open 
the possibility for analytics to offer insight into epistemic beliefs, as summarized in Table 2, which 
returns to the broad epistemic "dimensions" introduced earlier (Table 1), and gives examples of proxies 
within socio-semantic trace data, plus guidance indicative of the sorts of challenges that might be posed 
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to students to extend their epistemic cognition and probe their learning processes. 4 

We emphasize again that is not just the representational affordances of the analytics tool, but its use 
that will ultimately determine its impact. The representational scheme of semantically typed nodes and 
links clearly requires learner reflection and intentional artefact construction not normally demanded by 
conventional social platforms, and is thus theoretically tied to a socio-culturally inspired triad. However, 
both the constructs and the trace should be seen in their situated context: prior research into 
argumentation demonstrates that many elements in the cognitive and socio-cultural environment 
influence the usage patterns and impact of such tools (Scheuer, Loll, & McLaren, 2010). Key variables 
include user familiarity with the tool, their willingness to challenge each other in public, and the task set 
by teachers (this is a reiteration of the claims made at the end of Section 2). Thus, these should be 
dynamic tools, and empirical work will be needed to explore the relationship between feedback given, 
representations allowed, student responses to feedback, and the impact of this on learning. 


Table 2. Trace Patterns and Guidance for Epistemic Beliefs 


Dimensions of 
Epistemic Belief 

Candidate Socio/Semantic Trace 
Patterns 

Guidance/Challenge 
to the Learner 

Certainty 

The degree to which 
knowledge is 
conceived as stable 
or changing, ranging 
from absolute to 
tentative and 
evolving knowledge 

Frequency and density of positive and 
negative polarity links (e.g., supports; 
builds on; is consistent with, versus 
challenges; refutes; is inconsistent with). 

Presence of stability markers — (e.g., 
current sources; bibliographic references; 
geographical spread). 

Frequency and density of meta-discourse 
markers in the text of nodes, which 
moderate assertions in scholarly ways 
(e.g., in contrast to; remains an 
unresolved issue ) 

Are there no downsides to this 

claim? 

Can you find a counterexample? 

Is this idea consistent across 
time/place? Have you looked at 

XY map? 


4 Following previous work (De Liddo, Buckingham Shum, Quinto, Bachler, & Cannavacciuolo, 2011) the basic 
analytic statistic is constructed as a percentage representation of the target type, over the total types created by 
the user. For example, the number of "opinion" nodes created, as a percentage of the total number of nodes 
created by that user. 
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Simplicity 

Standard network analytics describing 

Have you considered how X and Y 

The degree to which 

the topography of the structure 

might be connected? 

knowledge is 

Semantic analysis of node and link types 

Why is this node so 

conceived as 

describing the variety of types used by a 

(dis)connected? 

compartmentalized 
or interrelated, 
ranging from 

learner, the balance in link polarity 
(positive/neutral/negative), and the 
balance of propositional (causes; refutes; 

Would you not expect this 
network to be better connected? 

knowledge as made 

solves ) versus analogical thinking (is a 

Does it make sense that this 

up of discrete and 

metaphor of; is analogous to). 

paper is in the hub? 

simple facts to 

Semantic analysis of tags on nodes and 

Does this analogy hold in all 

knowledge as 
complex and 
comprising 
interrelated concepts 

links 

Encapsulation of node clusters or sub¬ 
networks within a broader category 
expressing a higher order construct 

cases? (And many other critical 
questions used to probe 
arguments: Buckingham Shum 
and Okada, 2008) 


The use of different argumentation 

Is it significant that these two 


schemes, which reflect different sources 

analysts are not connecting their 


of authority (argumentfrom expertise; 
argument by analogy; argument from 
precedent) 

ideas to each other? 


In collaborative maps, the social network 
view showing who is connected to whom, 
but what kinds of links 


Source 

Presence within the text of nodes of "1 

What do you think of these 

The relationship 

think" or other restatements of fact 

ideas? 

between knower and 

Challenges to claims through negative 

Doesn't your perspective raise 

known, ranging from 

link types and meta-discourse markers in 

important questions for this 

the belief that 

node text 

claim? 

knowledge resides 
outside the self and is 
transmitted, to the 

Few additional nodes made other than 
those created as quotations. 

How does the evidence relate to 
your view? 

belief that it is 
constructed by the 



self 
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Justification 

Judgements of relevance, and supporting 

What evidence do we have for 

What makes a 
sufficient knowledge 

or explanatory notes ("this 
evidences/explains x"); ties to method 
"ideas" 

this idea? 

Is it "good" evidence? Why/why 

claim, ranging from 

not? 

the belief in 
observation or 
authority as sources, 
to the belief in the 

Use of well known Argumentation 

Schemes, especially evidence and 
arguments grounded in the attribution of 
authority to people ("experts") or 

You seem to quote/cite sources 
without much commentary of 
your own — bring in your voice 

more. 

use of rules of inquiry 

publications based on their status 

and evaluation of 

("valuing peer review") 


expertise 

Rhetorical constructions in node text that 

show a critical use of sources 



This example thus provides space to consider the role of the epistemology, assessment, and pedagogy in 
bounding learning analytics. In this case, assessment — or "measurement," analytics, data gathering, 
etc. — is used to support learning, and the exploration of that learning. The technological support 
provides space for discursive, socio-culturally motivated pedagogy. Finally, the aim of the analytic device 
is not to make claims about a learner's knowledge states at any particular moment, nor to provide 
psychometric assessment of groups of students, but rather to provide a discussion object, and to probe 
student sense-making processes, and understanding — their justifications for claims, a classic definition 
of knowledge — and thus an epistemological component of our analysis. 

6.1 Many Lenses on Epistemic Beliefs 

Table 2 thus proposes one set of traces from which meaningful data could be captured. This is not, 
however, to dismiss other approaches discussed in Section 4.1. The epistemological approach discussed 
throughout this work is instead intended to indicate that what drives our learning analytics — and 
assessment — is not what they are, but rather, what we do with them. Our suggestion is that many of 
these approaches to learning analytics — these dirty lenses on the world — provide insights into 
different levels of learning, and tools for meaning-making. For example, with this richer than normal 
data model in place, it is very simple, computationally, to feed back the number of ideas and connection 
types used, but this may provoke meaningful dialogue regarding what these other types might be used 
for, or why they have not thus far been used. Similarly, constructive discourse might occur around the 
reasons why one student's map is more connected (but perhaps not appropriately so) than another's. 

There is a strong relationship between analytics, assessment, pedagogy, and epistemology (Figure 1); 
learning analytics should be mindful of this triad, which socio-cultural analytics bridges well. Our 
approach should be seen as one of "many lenses" for many contexts, used in combination with the more 
conventional forms of learning analytics currently dominating. In this last section before concluding, we 
outline how the approaches discussed in Section 4.1 relate to epistemic beliefs, and some strengths and 
limitations of these approaches: 

Learning analytics based on Transactional approaches. Approaches that emphasize fixed, "correct" 
knowledge, over how those facts are used to display understanding, are likely to encourage lower 
epistemic cognition, and implicate more "realist" epistemologies that see knowledge as a reflection of 
"things" in the world. 
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Learning analytics based on Constructivist approaches. Similarly, there may be an overemphasis on a 
limited range of knowledge in constructivist approaches that emphasize development qua progression, 
but without considering the socio-cultural context in which that progression occurs, nor the wide range 
of uses for which it may be deployed. This may be particularly true in constrained systems that guide 
students through pre-set tasks and levels of attainment to meet, pre-specified software, and so on, as 
compared to those exploring knowledge co-constructed in discourse Understanding the ways that 
students build knowledge claims — understanding connections, justifications, change over time, and 
nuance — is fundamental to understanding their epistemic beliefs. Knowing that a student is at stage x 
of y in development may be less significant. 

Learning analytics based on Apprenticeship approaches. In a similar vein, apprenticeship approaches 
can offer useful insight into group membership and the development of a student's thinking. However, 
the approach described in this paper suggests the best way to think about such approaches is with 
respect to the functional role that such community membership plays in a student's epistemic action, 
and their normative standards. 

Learning analytics based on Subjectivist approaches. Learning analytics based on "affect" could be 
useful to the analysis of epistemic beliefs, with their analysis of "satisfaction" with information, e.g., 
enquiry-based learning (Ferguson et al., 2011); self-efficacy in IR information seeking (Tsai & Tsai, 2003); 
satisfaction with search results (Huffman & Hochster, 2007). As such, affective analytics might be used 
to explore whether learners are prematurely satisfied with findings that a peer or educator deems to be 
inadequate, or if they have an appropriate sense of disquiet or frustration with a flawed argument or 
methodology. 

7. CONCLUSION 

This paper opened with the argument that the triad of epistemology, assessment, and pedagogy are 
fundamentally entwined. Furthermore, we suggested that a focus on high-stakes assessment — which 
learning analytics may well be used to perpetuate — is detrimental to the wider enterprise of education, 
prioritizing the reliability of tightly defined assessments over continuing, formative assessment for 
learning, and authentically situated learning that is harder to fit into formal examination contexts. This is 
problematic as it limits the ways we can challenge students in assessments, and fails to reflect their 
encounters with knowledge claims in the world beyond the classroom walls. Learning Analytics should 
be mindful of the epistemological, assessment, and pedagogic implications of their application, and 
should be cautious of falling into technological determinism. 

We have highlighted that transactional approaches may emphasize the use of facts; constructivist the 
broad (and contextual) application of skills; subjectivist the self-efficacy and motivators of students; 
apprenticeship the dynamic practical-based learning that may occur through high-level membership of 
communities of practice; connectivism the ability of students to build up, link, and curate their 
knowledge "networks." A socio-cultural, pragmatic approach may offer an additional toolset, alongside a 
theoretical frame through which to use other learning analytics lenses. All are partial (in bias, and hence 
in their coverage of all that might be measured), but may be used in complementary ways. 

Analytics from user traces provide a means to track and record previously ephemeral process data, 
which could benefit assessment for learning in significant new ways. Pragmatist approaches, which 
emphasize use and meaning-making over the accrediting of true statements, may have an important 
role here. The grasp of curriculum facts and methods remains critical but the emphasis shifts to their 
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effective, contextualized use, in argument structures, in discussion, in problem-solving. A focus on the 
socio-cultural learning system draws attention to how analytics take into account the centrality of 
discourse for sense-making, and in constituting "context." 

We have gone beyond "our learning analytics are our pedagogy" (Buckingham Shum, 2012), arguing in 
more detail that analytics also embody related epistemological assumptions, thus perpetuating related 
assessment regimes. Since learning analytics unavoidably embody these triadic assumptions, there is the 
risk of perpetuating the limitations of current educational practices, especially those associated with 
high stakes testing. 

In conclusion, the stakes are high. As du Gay and Pryke (2002, pp. 12-13) observe, "accounting tools ... 
do not simply aid the measurement of economic activity, they shape the reality they measure." As 
educational institutions adopt platforms and applications that make it increasingly easy to capture 
particular kinds of data, conduct certain analyses, and foreground particular patterns for human 
attention, they reinforce their location within the triad. Educators, possibly held to account with 
analytics-based performance indicators, may well come to design learning in a way that will enable them 
to evidence impact with these new tools. Students, aware that their behaviour is being tracked, may 
similarly seek to "evidence" their progress, and game the more simplistic measures. Technology alone 
does not determine practice; moreover, as with any tool, it is not only the design of the tool, but the 
way in which it is wielded in context, that defines its value. 
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